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INTRODOCTION 


In  the  past  decade,  a  decision  aiding  technology  has  emerged  from 
the  discipline  of  artificial  intelligence  that  has  the  potential  for 
greatly  improving  the  decision  speed  and  quality  of  decision  makers 
information  analysis  and  problem  solving  process.  Specifically, 
rule-based  expert  systems  have  been  developed  which  exhibit  domain 
specific  problem  solving  behavior  that  is  comparable  in  quality  to 
that  shown  by  human  experts.  From  a  cognitive  human  factors 
perspective,  it  has  been  argued  and  generally  accepted  that  a 
prerequisite  for  effective  user/expert  system  interaction  is  a  high 
degree  of  consistency  between  the  user's  and  expert  system's  domain 
knowledge.  That  is,  the  expert  systems  should  solve  the  problem  in 
the  same  manner,  applying  the  same  rules,  that  a  human  expert  does. 

In  this  paper,  we  argue  that  this  argument  is  incorrect  and 
furthermore  that  for  many  contexts,  significant  consistency  in  the 
user  and  expert  systems  problem  solving  approach  is  actually 
undesirable.  Three  experiments  providing  empirical  support  for  this 
hypothesis  are  presented. 

Historically  most  expert  systems  have  been  developed  to  operate 
in  time-relaxed  consultation  environments  where  the  system  requests 
from  the  user  a  substantial  amount  of  information  about  the  problem  at 
hand.  Examples  of  fully  developed  systems  of  this  type  are  MYCIN 
(Shortliffe,  1976)  which  diagnoses  bacterial  infections  and  recommends 
antibiotic  therapy,  and  PROSPECTOR  (Duda,  et  al.,  1979)  which 
evaluates  geological  mineral  sites  for  potential  deposits.  Recently, 
however,  expert  system  technology  has  focused  on  the  use  of  expert 
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systems  as  intelligent  interfaces  between  a  user  and  a  larger  complex 
information  processing  system.  These  expert  interface  systems  use  the 
same  rule-base  program  architecture  found  in  consultation  systems, 
however,  they  automatically  acquire  data  from  an  external  source 
rather  than  requiring  data  information  from  the  user.  Thus  the 
primary  function  of  the  expert  interface  system  is  to  enhance  a  user's 
ability  to  utilize  an  external  data  source,  often  a  large  system  that 
operates  independently  of  tho  expert  interface.  Proposed  applications 
of  expert  interface  systems  include  centralized  sensor  integration  and 
display  control,  and  real  time  decision  making  support  (Walker 
and  Lehner ,  1985). 

From  a  human  factors  perspective,  there  are  some  key  aspects  of 
expert  interface  systems  that  make  them  very  different  from  expert 
consultation  systems.  First,  consultation  systems  tend  to  address 
problem  domains  with  a  well-established,  well-documented,  and  static 
knowledge  base.  Expert  interface  systems  on  the  other  hand,  tend  to 
involve  ill-specified  knowledge  bases,  where  human  experts  differ 
considerably  in  their  opinions.  Second,  the  time  constraints  on  the 
decision  processes  of  interface  systems  is  typically  shorter  than  is 
the  case  with  consultation  systems.  Third,  consultation  systems  are 
stand  alone,  requiring  the  user  to  enter  all  problem  specific  data. 
As  a  result,  consultation  system  users  must  have  sufficient  domain 
expertise  to  answer  the  system  queries.  In  contrast,  expert  interface 
systems  are  usually  embedded  within  a  larger  'background'  system. 
Thus,  users  are  not  a  priori  familiar  with  the  specifics  of  the 
problem  being  addressed.  Indeed,  the  user  may  not  even  know  a  problem 
exists  until  the  expert  system  has  already  analyzed  data  obtained  from 


the  background 
recommendations . 


system  and  generated 


its 


conclusions 


and 


The  user,  therefore,  will  typically  find  it  necessay 
to  examine  the  data  and  rules  that  led  to  the  system  conclusions. 

Given  the  above  types  of  expert  systems,  the  user/expert  system 
interaction  can  be  viewed  as  a  situation  where  two  problem  solvers  are 
trying  to  cooperatively  solve  a  common  decision  problem  despite  the 
fact  that  these  two  experts  may  use  different  decision  process, 
heuristics  and  data  to  solve  the  common  problem.  Two  cognitive 
factors  seem  particularly  relevant  when  examining  this  interaction: 

(1)  the  degree  to  which  the  user  and  expert  system's  problem  solving 
processes  utilize  similar  domain  and  problem  specific  knowledge,  and 

(2)  the  user's  conceptual  understanding  of  the  basic  principle  of  the 
system's  problem  solving  processes,  i.e.,  the  user's  mental  model.  It 
is  postulated  that  in  a  user  and  expert  system  problem  solving 
situation,  performance  is  dependent  on  both  of  these  two  variables, 
and  that  there  is  a  strong  interaction  between  them  in  their  impact  on 
performance . 

Specifically,  we  hypothesize  that  a  good  mental  model  of  expert 
system  processing  would  facilitate  user  understanding  of  system 
recommendations  and  explanations  even  if  the  system's  problem  solving 
approach  is  substantially  different  from  the  user's.  Furthermore, 
when  the  user  and  expert  system  are  viewed  as  two  interacting 
production  systems,  the  potential  improvement  of  cooperative  problem 
solving  over  individual  user  or  system  problem  solving  increases 
proportionately  as  the  degree  of  overlap  between  the  user's  and  the 
expert  system's  problem  solving  processes  decreases,  i.e.,  with 


decreasing  cognitive  consistency 


However,  in  order  to  realize  this 


potential,  the  user  must  be  able  to  incorporate  the  system's 

cognitions  into  his  or  her  own  reasoning  about  the  problem.  Thus  it 
is  predicted  that  when  a  user  possesses  an  accurate  mental  model  , 
cognitive  inconsistency  should  generally  result  in  better  performance 
than  problem  solving  involving  a  high  level  of  cognitive  consistency. 
If  the  user,  on  the  other  hand,  does  not  have  a  good  mental  model  of 
the  expert  system's  processing,  the  conventional  wisdom,  suggesting 
that  performance  improves  as  the  overlap  between  the  user's  and 
sytem's  cognitions  increase,  may  be  correct.  The  following  three 
experiments  test  these  general  hypotheses. 


• .  • .  •V  < 

■  - 1 

:•>**> 


EXPERIMENT  1 


Method 


In  this  experiment,  both  the  subjects  and  expert  system  had 


isomorphic  decision  rules  (i.e.,  they  would  provide  the  same 
solution),  however,  there  was  inconsistency  in  the  data  sets.  The 
expert  system  had  access  to  data  which  the  subjects  initially  did  not 
have,  while  the  data  the  subjects  did  possess  was  more  accurate  than 


the  expert  system's  data.  Under  these  conditions,  subjects  were 
required  to  interact  with  the  system  to  obtain  all  relevant  data, 
however,  the  expert  system  did  not  necessarily  generate  the  correct 
solutions . 

Subjects .  Thirty-two  (16  male  and  16  female)  undergraduate 
students  from  the  Catholic  University  of  America  served  as  volunteer 


subjects  in  this  study.  The  mean  age  was  19.3  years  with  a  range  of 


17-22  years.  None  of  the  subjects  had  previous  experience  with 
rule-based  systems  or  computer-aided  problem  solving  tasks. 

Materials.  The  framework  used  for  development  of  the  intelligent 
interface  was  ERS,  Embedded  Rule  Based  System  (Barth,  1984),  which  is 
in  many  respects  similar  to  the  well  known  PROSPECTOR  inference  engine 
(Duda,  et  al.,  1979).  The  ERS  system  consists  of  an  inference  engine, 
rule  base  parser,  and  language  for  representing  rules.  Rules  in  a 
test  file  are  parsed  and  compiled  into  internal  data  structures  during 
run-time  initialization.  The  inference  engine  then  uses  these  data 
structures  to  drive  the  system's  decision  making  process.  This 
process  may  involve  gathering  evidence  from  the  user,  as  is  usually 
done  in  expert  consultation  systems,  or  from  a  set  of  primitives 
supplied  for  a  particular  application,  or  both.  As  sufficient 
evidence  is  gathered,  conclusions  or  advice  is  reported  in  the  form  of 
the  degree  of  belief  in  the  top  level,  goal  hypotheses  that  were 
defined  in  the  rule  base.  The  system  continues  gathering  evidence  and 
reporting  advice,  until  no  more  evidence  remains  to  be  gathered,  or 
the  user  issues  a  quit  command.  Written  in  Pascal,  ERS  was  developed 
on  a  VAX-11/780  under  the  UNIX  operating  system,  version  4.1  bsd .  and 
has  been  installed  on  an  IBM  PC  with  128  bytes  of  memory,  under  the 
UCSD  P-System. 

For  this  study,  a  simplified  version  of  ERS  was  implemented  on  an 
Apple  lie  microcomputer  with  64  bytes  of  memory.  The  rule  base 
representing  the  testbed  domain  consisted  of  an  inference  network 
containing  63  nodes,  5  goal  hypotheses,  39  rules,  and  109  links 
between  nodes.  This  system  was  employed  for  all  three  experiments. 

Experimental  Design.  A  2  x  2  factorial  design  was  used  to  create 
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backward-chaining  manner.  Low  cognitive  consistency  occurred  when  the 
user  problem  solved  in  an  almost  opposite,  data-driven, 
forward-chaining  process.  The  application  of  both  procedures  resulted 
in  identical  final  soutions  for  all  data  possibilities. 

The  two  levels  of  the  second  independent  variable,  mental  model, 
were  (1)  accurate  mental  model  and  (2)  no  mental  model.  Subjects  in 
the  accurate  mental  model  condition  received  as  part  of  their 
instructions  a  written  description  of  an  inference  network.  This 
section  described  the  structure  of  a  general  inference  network, 
explained  how  the  expert  system  identified  goals,  intermediate 
hypothesis,  and  data  items,  and  chained  up  and  down  the  network  to 
obtain  degree  of  certainty  values  for  each  goal.  Included  in  this 
section  was  a  pictorial  display  of  an  inference  net  and  a  simple 
example  of  its  operation.  By  working  through  this  section,  the  user 
developed  a  mental  model  of  how  the  expert  system  solved  problems. 

Testbed  Doman.  The  experimental  domain  for  all  three  experiments 
was  a  simulated  stock  market  setting  in  which  five  different  types  of 
securities  fluctuated  in  value  during  the  testing  sessions.  The 


increase  or  decrease  in  security  values  was  influenced  by  two  types  of 
data:  (1)  general  market  conditions,  and  (2)  specific  trading 


activities.  Market  conditions  concerned  the  degree  to  which  the 
general  market  state  could  be  identified  as  "bear,"  "mixed,"  or 
"bull."  Trading  activities  described  the  volumes  of  buying/selling 
during  a  hypothetical  time  frame,  e.g.,  "Blue  Chip  securities  were 
sold  by  5000  shares  in  two  weeks." 

Task  problems  were  constructed  by  creating  patterns  of  specific 
market  conditions  and  trading  activities  and  defining  the  resultant 
security  value  fluctuations.  For  each  possible  pattern  of  data 
combinations  there  was  exactly  one  of  the  five  securities  whose 
value  increased  the  highest.  Thus,  for  each  task  problem  there  was 
one  optimum  security  that  should  be  recommended  for  purchase. 

Expert  System.  The  expert  system,  used  in  the  three  experiments, 
included  an  inference  network  in  which  each  of  the  five  securities  was 
set  as  a  goal.  The  experimenter  provided  the  system  with  access  to 
data  concerning  previous  market  conditions  (e.g.  "bear"),  current 
market  conditions  (e.g.,  "mixed"),  and  the  volumes  and  direction  that 
each  security  is  currently  being  traded.  The  system  utilized 
backward-chaining  procedures  to  validate  the  degree  of  belief  in  each 
of  the  lowest-level  data  nodes,  assess  value  to  the  intermediate 
hypotheses,  and  in  turn  estimate  the  degree  to  which  each  of  the  five 
securities  should  be  recommended  for  purchase.  At  this  point  the 
system  displayed  the  five  securities  as  rank-ordered  (greatest  to 
poorest)  recommendations  for  purchase. 

Procedure.  After  reading  a  description  of  the  experiment  and 
signing  a  consent  form,  subjects  received  the  instruction  booklet 
pertaining  to  their  particular  mental  model/cognitive  consistency 
group  condition.  These  instructions  specified  the  objectives. 
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procedures,  and  requirements  of  the  problem  solving  task.  Each  set  of 
instructions  previously  defined  the  level  of  mental  model  and  type  of 
problem  solving  procedures  to  be  implemented. 

Subjects  were  seated  at  a  large  table  directly  in  front  of  the 
expert  system  with  ample  space  to  arrange  their  individual  problem 
solving  sheets.  Upon  completion  of  the  experimental  booklet,  subjects 
received  three  test  problems  to  practice  use  of  their  own  procedures. 
The  type  of  data  presented  to  the  subject  was  of  the  same  nature, 
i.e.,  market  conditions  and  trading  activities,  that  the  expert  system 
utilized.  Competent  use  of  the  appropriate  styles  of  procedures, 
either  (1)  cognitive  consistent  or  rule-driven,  backward  chaining,  or 
(2)  cognitive  inconsistent  or  data-driven,  forward  chaining,  was 
reached  by  all  participants. 

At  this  time  subjects  received  practice  interacting  with  the 
expert  system.  After  viewing  the  system's  prioritized  list  of 
security  recommendations,  users  practiced  querying  the  system  for  its 
decision  rationale.  In  other  words,  the  user  saw  what  the  expert 
system  recommended  for  a  specific  problem  and  sought  to  determine 
"how"  and/or  "why"  this  particular  recommendation  was  reached. 
Through  the  use  of  a  node  description  command,  initiated  by  the 
entering  of  a  "d,"  carriage  return,  and  specification  of  the  node  name 
to  be  examined,  users  were  able  to  examine  important  components  of  the 
systems  logic  and  reasoning.  Successful  interaction  with  the  system 
involved  the  specification  of  several  successive  node  description 
commands  to  reach  meaningful  information,  e.g.,  the  degree  of 

certainty  of  the  systems  lowest  level  data  items. 

The  experimental  process  tested  subjects  individually  for  six 


separate  problems.  A  time  constraint  of  150  seconds  was  imposed  upon 
each  task.  Pilot  studies  demonstrated  this  time  allowance  as  adequate 
for  proficient  interaction  with  the  system  and  use  of  individual 
problem  solving  procedures. 

Each  of  the  problem  solving  tasks  proceeded  as  follows.  Seated 
in  front  of  the  expert  interface  system  with  individual  problem 
solving  sheets  within  easy  reach,  a  subject  viewed  the  system's  rank 
ordered  list  of  security  recommendations  for  the  current  problem.  The 
experimenter  handed  the  subject  written  data  concerning  "previous 
market"  and  "current  market"  estimations.  At  this  point  the  subject 
could  allocate  time  to  either  querying  the  system  with  the  "d" 
command,  utilizing  the  individual  problem  solving  sheets,  or  a 
combination  of  both.  After  180  seconds,  the  subject  viewed  the 
system's  recommendations  a  final  time  and  terminated  the  problem 
solving  session.  The  subject  recorded  on  an  answer  sheet  by  simply 
checking  off  the  one  or  two  securities  most  recommended  and  by  writing 
a  few  sentences  describing  why. 

At  the  completion  of  six  such  problems,  subjects  completed  a 
brief  questionnaire  assessing  reaction  to  the  problem  solving 
experience  and  were  adequately  debriefed. 

Performance  Measures.  For  each  of  the  six  problem  solving  tasks, 
one  of  the  five  securities  had  been  evaluated  as  the  optimal 
recommendation  during  task  construction.  This  benchmark  solution  was 
reached  by  utilizing  problem  solving  procedures  applied  to  the 
complete  data  set.  It  should  be  noted  that  both  types  of  procedures, 
forward-chaining  and  backward  chaining,  reached  the  same  conclusions 
given  the  same  data.  Thus,  for  each  of  the  six  problems  a  benchmark 
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solution  was  constructed  as  a  comparison  for  the  subjects'  responses. 

The  major  performance  measurement  was  the  number  of  times  a 
subject's  response  matched  the  predetermined  optimal  one.  Individual 
scores  could  range  from  0,  none  of  the  problems  correct,  to  6,  all  of 
the  solutions  matched  with  the  optimal  ones. 

A  second  performance  measure  was  the  10-item  subjective 
questionnaire.  Subjects  indicated  on  a  10-point  scale  from  0  ("very 
strongly  disagree")  to  10  ("very  strongly  agree")  their  agreement  with 
statements  addressing  (1)  the  understanding  of  the  expert  system's 
operating  procedures,  (2)  the  ease  of  system  use,  (3)  the  confidence 
of  final  user  decisions,  and  (4),  the  adequacy  of  the  time  allotment. 

Finally,  the  number  of  user  queries  to  the  expert  interface 
system  were  recorded.  These  "d"  commands  were  noted  for  each  subject 
over  each  of  the  six  problems. 

Results 

The  principle  issue  in  this  experiment  was  the  combined  effect  of 
mental  model  and  cognitive  consistency  on  subjects  problem  solving 
with  an  expert  system.  The  data  set  was  subjected  to  a  2  x  2  analysis 
of  variance  procedure  investigating  the  main  effect  of  each  of  the 
independent  variables  a  well  as  the  interaction  between  them.  The 
strong  main  effect  of  mental  model,  F(l,28)  =  11.15,  p  =  .00214, 
demonstrated  that  an  effective  understanding  of  the  system's  operating 
procedures  facilitated  cooperative  problem  solving  quality.  As 
predicted,  a  main  effect  for  cognitive  consistency  did  not  occur, 
F(l,28)  =  0.0.  The  presence  of  a  significant  interaction  between 
mental  model  and  cognitive  consistency,  F(l,28)  =  8.2,  p  =  .0079, 
supports  our  central  hypothesis  and  theoretical  basis  for  user/system 


problem  solving. 


A  graphical  presentation  of  subject  correct  responses  as  a 
percentage  of  total  problems  is  depicted  in  Figure  1.  The  mental 
model/cognitive  consistency  interaction  is  easily  observable. 
Individual  comparisons  confirmed  several  hypotheses.  For  those 
possessing  an  accurate  mental  model,  inconsistent  (forward-chaining) 
procedures  led  to  a  significantly  greater  performance,  than  consistent 
(backward-chaining)  procedures  t(14)  =  2.17,  p  <  .05.  For  users 

without  an  accurate  mental  model  performance  improved  when  consistent 
(backward-chaining)  procedures  were  followed,  although  this  difference 
failed  to  reach  significance,  t(14)  =  1.90,  p  =  .079.  When  evaluating 
the  two  groups  implementing  inconsistent  procedures,  subjects 

possessing  an  accurate  mental  model  performed  significantly  better, 
t(14)  =  4.13,  p  <  .01. 

Further  data  analysis  was  performed  by  evaluating  subjects 
reponses  to  the  10-item  subjective  questionnaire.  Users  receiving 
accurate  mental  models  reported  greater  "understanding  of  the  system's 
operating  procedures,"  means  of  5.7  and  5.7  (cognitive  inconsistent 
and  cognitive  consistent  respectively)  than  those  without  an  accurate 
mental  model,  means  of  3.4  and  3.8  respectively.  Reports  of  "ease  of 
system  use"  followed  the  general  interaction  pattern,  the  means  being 
7.9,  6.7,  6.8,  and  5.8.  "Confidence  of  final  user  decisions"  followed 
the  same  pattern  with  means  of  7.9,  6.7,  6.8,  and  5.8.  The  "adequacy 
of  the  time  allotment"  revealed  the  lowest  performing  group  of  no 
mental  model/cognitive  inconsistent  to  be  most  time  pressured,  mean  of 
2.8,  compared  to  the  other  three  groups  with  means  of  4.8,  5.5,  and 
4.4  respectively. 
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FIGURE  ^ 

PERFORMANCE  AS  PERCENTAGE  OF  OPTIMAL  RESPONSES  BY  GROUP 

(Experiment  1) 
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The  final  performance  measure  of  the  number  of  user  queries 
during  user/system  interaction  revealed  no  differences  across  the  four 


conditions,  the  means  being  4.9,  5.2,  5.27,  and  5.01  respectively. 


EXPERIMENT  2 


Method 


In  Experiment  1,  there  was  a  60%  difference  in  performance 
between  the  good  and  no  mental  model  groups  under  the  cognitive 
inconsistent  condition.  One  hypothesis  to  account  for  this  difference 
is  that  good  mental  model  subjects  were  much  more  facile  in 
manipulating  the  expert  system  than  the  no  mental  model  subjects.  As 
a  result  they  were  able  to  manipulate  the  system  to  get  access  to  all 
relevant  data,  while  the  no  mental  model  subjects  were  not  able  to  qd 
this.  Experiment  2  tested  this  hypothesis.  Specifically,  this 
experiment  examined  whether  users  utilizing  forward-chaining 
(cognitive  inconsistent)  procedures  with  no  mental  model  of  the  expert 
system's  problem  solving  processes  could  significantly  improve  their 
problem  solving  performance  if  given  direct  access  to  the  system's 
data.  It  was  predicted  that  if  users  with  no  mental  model  using 
inconsistent  procedures  could  obtain  an  immediate  display  of  all 
relevant  data  the  system  had  available,  their  performance  would  not 
significantly  differ  from  users  with  a  good  mental  model  using 
inconsistent  procedures. 

Subjects.  Sixteen  (17  male  and  9  female)  undergraduate  students 
form  the  George  Mason  University  served  as  subjects  in  this  study. 


T 


The  mean  age  was  18.5  years  with  a  range  of  17-21  years.  None  of  the 
subjects  had  previous  experience  with  rule-based  systems  or 
computer-aided  problem  solving  tasks. 

Experimental  Design.  A  between  subjects  design  was  used  to 
examine  one  independent  variable,  mental  model,  with  respect  to  low 
cognitive  consistency.  The  two  levels  of  mental  model,  were  (1) 
accurate  mental  model  and  (2)  no  mental  model.  Subjects  in  the 
accurate  mental  model  condition  received  as  part  of  their  instructions 
a  written  description  of  an  inference  network.  This  was  the  same 
description  as  that  given  to  subjects  in  Experiment  1.  The  subjects 
in  the  no  mental  model  condition  were  not  given  this  description  of 
the  system's  inference  network,  however,  they  could  access  the 
system's  data  by  using  a  data  description  command  as  described  in  the 
Procedure  section. 

As  in  Experiment  1,  low  cognitive  consistency  was  created  in  all 
subjects  by  teaching  the  user  to  problem  solve  in  a  data-driven 
forward-chaining  process. 

Procedure.  The  procedure  was  identical  to  that  of  Experiment  1, 
with  two  exceptions.  The  first  being  that  users  with  a  good  mental 
model  and  inconsistent  procedures  were  compared  only  to  users  with  no 
mental  model  and  inconsistent  procedures.  Thus,  two  groups  of 
subjects  (rather  than  four  groups  as  in  Experiment  1)  were  examined. 

The  second  difference  between  Experiment  1  and  Experiment  2  was 
that  a  data  desription  command  was  employed  in  Experiment  2.  Users 
with  no  mental  model  were  able  to  examine,  through  the  use  of  a  data 
description  command,  the  system's  actual  data  (i.e.,  the  market 
conditions  and  trading  activities)  that  it  used  to  reach  its 


conclusions.  This  command  was  initiated  by  entering  a  "w"  ,  carriage 
return  and  specification  of  the  data  to  be  examined. 

Performance  Measures.  The  performance  measures  were  identical  to 
those  of  Experiment  1  with  the  exception  that  the  number  of  user 
queries  to  the  expert  interface  system  were  not  recorded. 

Results 

This  experiment  examined  the  performance  of  users  without  an 
accurate  mental  model  and  with  direct  access  to  the  system's  data 
(i.e.,  the  data  description  command)  with  the  performance  of  users 
with  an  accurate  mental  model  and  no  direct  access  to  the  system's 
data.  The  data  set  was  subjected  to  a  t  test  (one  factor  analysis  of 
variance)  procedure.  Results  demonstrated  that  users  with  direct 
access  to  the  system's  data  but  without  an  accurate  mental  model  did 
not  significantly  differ  from  users  without  direct  access  to  the 
system's  data  but  with  a  good  mental  model,  F(1,14)  =  .78,  p  >  .05.  A 
graphical  presentation  of  subjects'  correct  responses  as  a  percentage 
of  total  problems  is  depicted  in  Figure  2. 

Further  data  analysis  was  performed  by  evaluating  subjects' 
responses  to  the  10-item  subjective  questionnaire.  As  expected,  users 
receiving  accurate  mental  models  reported  greater  "understanding  of 
the  system's  operating  procedures"  than  those  without  an  accurate 
mental  model  (means  of  6.3  and  3.7  respectively).  Users  with  accurate 
mental  models  also  reported  greater  "confidence  of  their  final 
decisions"  than  those  users  without  an  accurate  mental  model  (means  of 
6.4  and  3.5  respectively).  The  "adequacy  of  the  time  allotment" 
followed  the  same  pattern  with  means  of  5.6  and  3.2.  However,  "ease 
of  system  use"  was  found  to  be  greater  in  users  with  no  mental  model 


than  in  users  with  an  accurate  mental  model  (means  of  5.2  and  4.8 


respectively).  This  last  finding  suggests  that  if  a  user  with  no 
mental  model  has  the  capability  to  access  the  system's  data  directly, 
he  or  she  will  find  the  system  much  easier  to  use. 


EXPERIMENT  3 


Method 

In  the  previous  two  experiments,  subjects  had  little  need  to 
actually  trace  the  expert  system's  reasoning  to  obtain  assistance  in 
problem  solving,  rather  they  simply  used  a  sequence  of  commands  to 
reach  the  system's  data.  Consequently,  it  was  not  clear  the  extent  to 
which  a  good  mental  model  helped  subjects  actually  understand  how  the 
system  generated  a  recommendation.  This  third  experiment  addressed 
this  latter  issue.  In  particular,  we  examined  the  effect  a  user's 
mental  model  has  on  performance  when  the  user  is  required  to  examine 
and  identify  inconsistencies  between  the  system's  rule-base  and  the 
user's  problem  solving  procedures.  It  was  predicted  that  a  good 
mental  model  would  allow  the  user  to  effectively  chain  up  and  down 
through  the  system's  rule-base,  whereas  users  with  no  mental  model 
would  in  effect,  become  "lost"  unless  they  were  using  the  same  problem 
solving  procedures  (i.e.,  backward  chaining)  as  the  system. 

Subjects .  Thirty-two  (14  male  and  18  female)  undergraduate 
students  from  the  Lord  Fairfax  Community  College  served  as  volunteer 
subjects  in  this  study.  The  mean  age  was  19.6  years  with  a  range  of 
17-30  years.  None  of  the  subjects  had  previous  experience  with 


rule-based  systems  or  computer-aided  problem  solving  tasks. 

Experimental  Design.  The  experimental  design  was  identical  to 
that  of  Experiment  1. 

Procedure .  The  procedure  was  identical  to  that  of  Experiment  1 
with  the  exception  that  subjects  received  a  statement  indicating  the 
system's  solution  if  it  was  free  of  error  in  addition  to  problem 
solving  sheets  that  led  to  solutions  inconsistent  with  the  rule-base 

employed  by  the  system.  The  system's  solution  if  it  was  free  of  error 

was  identical  to  that  obtained  using  the  problem  solving  sheets. 
Thus,  the  system  and  the  problem  sheets  resulted  in  two  different 
solutions.  Subjects  were  told  that  the  system  contained  an  error  in 
the  way  it  solved  each  problem.  Subjects  were  required  to  identify 
the  inconsistency  or  difference  between  their  problem  solving  sheets 
(or  the  problem  solving  logic  used  by  the  system  if  it  was  free  of 
error)  and  the  procedures  (or  problem  solving  logic)  utilized  by  the 
expert  system  (e.g.,  situation  A  on  Sheet  1  was  interchanged  with 
situation  C  on  Sheet  3) . 

Performance  Measures .  The  major  performance  measure  was  the 
number  of  times  a  subject  correctly  identified  an  inconsistency 

between  the  system's  solution  and  the  solution  provided  by  the  problem 
solving  sheets.  All  other  performance  measures  were  identical  to 

those  utilized  in  Experiment  1. 

Results 

The  principle  issue  in  this  experiment  was  the  combined  effect  of 
mental  model  and  cognitive  consistency  on  the  user's  ability  to 
identify  inconsistencies  between  the  system's  rule  base  and  the  user's 
problem  solving  procedures.  The  data  set  was  subjected  to  a  2  x  2 
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analysis  of  variance  procedure  investigating  the  main  effect  of  each 
of  the  independent  variables  as  well  as  the  interaction  between  them. 
The  strong  main  effect  of  mental  model,  F(l,28)  =  18.67,  p  =  .0002, 
demonstrated  that  an  effective  understanding  of  the  system's  operating 
procedures  facilitated  rule  inconsistency  identification.  As 

expected,  a  main  effect  for  cognitive  consistency  did  not  occur, 
F(l,28)  =  2.07,  p  >  .05.  The  interaction  between  mental  model  and 

cognitive  consistency,  F(l,28)  =  .52,  p  >  .05,  was  not  significant. 

A  graphical  presentation  of  subject  correct  responses  as  a 
percentage  of  total  problems  is  depicted  in  Figure  3.  The  mental 
model/cognitive  consistency  interaction  is  easily  observable.  For 
users  possessing  no  mental  model,  consistent  (backward-chaining) 
procedures  led  to  a  significantly  greater  performance,  than 
inconsistent  (forward-chaining)  procedure  t(14)  =  2.39,  p  <  .05.  For 
users  with  an  accurate  mental  model,  performance  did  not  significantly 
differ  for  the  two  types  of  problem  solving  procedures  t(14)  =  1.07,  p 
>  .05.  Users  possessing  an  accurate  mental  model  and  consistent 

procedures  performed  significantly  better  than  users  utilizing 
consistent  procedures  with  no  mental  model  t(14)  =  2.55,  p  <  .05. 

The  data  analysis  of  the  10-item  subjective  questionnaire 
revealed  results  supporting  the  performance  data.  Users  receiving 
accurate  mental  models  reported  "greater  confidence  of  their  final 
decisions",  means  of  7.0  and  6.75  (cognitive  inconsistent  and 
cognitive  consistent,  respectively)  than  those  without  an  accurate 
mental  model,  means  of  4.75  and  6.5  respectively.  The  "adequacy  of 
the  time  allotment"  revealed  the  lowest  performing  group  of  no  mental 
model/cognitive  inconsistent  to  be  the  most  time  pressured,  mean  of 
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FIGURE  3 

PERFORMANCE  AS  PERCENTAGE  OF  OPTIMAL  RESPONSES  BY  GROUP 

(Experiment  3) 


5.25  compared  to  the  other  three  groups,  means  of  6.0  for  the  no 
mental  model/cognitive  consistent  group,  6.4  for  the  accurate  mental 
model/cognitive  consistent  group,  and  7.0  for  the  accurate  mental 
model/cognitive  inconsistent  group. 

The  results  for  two  items  on  the  subjective  questionnaire  yielded 
somewhat  inconsistent  results.  Users  utilizing  consistent  procedures 
with  no  mental  model  reported  "greater  understanding  of  the  system's 
operating  procedures",  mean  of  7.21  followed  by  those  users  receiving 
a  good  mental  model  and  inconsistent  procedures,  mean  of  6.5.  The  no 
mental  model  group  receiving  inconsistent  procedures  had  a  mean  of 
6.1,  and  finally,  users  with  a  good  mental  model  and  consistent 
procedures  had  a  mean  of  5.1.  Greater  "ease  of  system  use"  was 
reported  by  the  accurate  mental  model/cognitive  inconsistent  group, 
mean  of  8.3,  followed  by  the  no  mental  model/cognitive  consistent 


group  with  a  mean  of  7.75.  These  group  means  were  followed  by  users 
receiving  a  good  mental  model  and  cognitive  consistent  procedures, 
mean  of  6.8,  and  finally,  the  no  mental  model/cognitive  inconsistent 
group,  mean  of  5.8. 

The  final  performance  measure  of  the  number  of  user  queries 
during  user/system  interaction  revealed  no  significant  differences 
across  the  four  conditions,  the  means  being  3.22,  3.33,  3.57,  and  3.65 
respectively. 


DISCUSSION  AND  CONCLUSION 


The  basic  conclusions  for  these  three  experiments  appears  to  be 


rows 


that  in  a  cooperative  human/intelligent  machine  problem  solving 
setting,  where  the  human  and  machine  employ  different  problem  solving 
procedures,  it  is  generally  essential  that  the  user  have  an  accurate 


model  of  how  that  machine  operates.  Even  for  relatively  simple 

decision  problems,  such  as  the  one  used  in  these  experiments,  a  poor 
mental  model  leads  to  anywhere  from  a  30  to  60%  drop  in  performance. 
For  complex,  real-world  expert  system  applications  therefore  a  good 
mental  model  may  often  be  a  necessary  condition  for  effective 
user/expert  system  interaction.  Indeed,  from  the  perspective  of  the 
practical  implications,  the  most  immediate  impact  is  what  the  results 
suggests  about  how  user  interactions  with  expert  consultation  and 
expert  interface  systems  will  differ.  Users  of  expert  interface 
systems  are  likely  to  be  significantly  inconsistent  from  the  expert 
system  in  both  the  problem  specific  data  they  are  initially  aware  of 
and  the  domain  specific  heuristics  utilized  in  problem  solving. 
Consequently,  user/expert  interface  system  interaction  is  a  situation 
that  naturally  reflects  a  great  deal  of  cognitive  inconsistency.  As  a 
result,  creating  an  accurate  mental  model  may  be  an  essential 
ingredient  for  the  successful  transfer  of  interface  system  to 
operational  use. 

Regarding  the  completeness  of  the  above  research,  it  should  be 
recognized  that  these  experiments  operationalized  cognitive 
consistency  as  the  match  between  the  user's  and  the  expert  system's 
procedures.  Other  dimensions  of  cognitive  consistency  need  to  be 
examined.  Furthermore,  a  node  description  command  was  the  only  type 
of  explanation  a  user  could  receive  in  this  study.  This  was  chosen 
primarily  because  of  the  imposed  time  constraint  and  the  nature  of  the 
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task  setting.  Other  explanation  capabilities  should  be  examined, 
including  a  rule-trace  or  presentation  of  the  system's  intermediate 
hypotheses.  Finally,  user  groups  of  diverse  expertise  levels  should 
be  studied  over  several  domains  and  under  a  varying  range  of  time 
constraints.  Ongoing  research  is  currently  addressing  futher  issues 
in  the  user/system  interface  in  an  attempt  toward  developing  a  more 
complete  set  of  empirically-tested  theoretical  principles  of 
user/expert  system  interaction. 
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